{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(code-advanced)=\n",
    "# Advanced Coding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "If you're just starting to code, you can safely skip this chapter.\n",
    "```\n",
    "\n",
    "## Introduction\n",
    "\n",
    "This chapter covers some more advanced programming concepts. It's not strictly necessary to master the content of the book, but it's here in case you want a deeper understanding or in case you find that you eventually need to draw on more sophisticated programming tools and concepts.\n",
    "\n",
    "This chapter has benefitted from the online book [*Research Software Engineering with Python*](https://merely-useful.github.io/py-rse/), the [official Python documentation](https://www.python.org/), the excellent [30 days of Python](https://github.com/Asabeneh/30-Days-Of-Python), and the [Hitchhiker's Guide to Python](https://docs.python-guide.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sets\n",
    "\n",
    "A set in coding is a collection of unordered and unindexed distinct elements (in analogy to the mathematical definition of a set). To define a set, the two commands are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st = {}\n",
    "# or\n",
    "st = set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These aren't very interesting though! Here's a set with some values in:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people_set = {\"Robinson\", \"Fawcett\", \"Ostrom\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can we do with it? We can check its length using `len(people_set)` and we can ask whether a particular entry is contained within it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"Ostrom\" in people_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add multiple items or another set using `.update` or `.union`, or a single item using:\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people_set.add(\"Martineau\")\n",
    "people_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can remove entries with `.remove(entry_name)` or, to remove only the last entry `.pop()`. You can easily convert between lists and sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(people_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The real benefits of sets are that they support set operations, though. The most important are `intersection`,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st1 = {\"item1\", \"item2\", \"item3\", \"item4\"}\n",
    "st2 = {\"item3\", \"item2\"}\n",
    "st1.intersection(st2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`difference`,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st1 = {\"item1\", \"item2\", \"item3\", \"item4\"}\n",
    "st2 = {\"item2\", \"item3\"}\n",
    "st1.difference(st2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and symmetric difference,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st1 = {\"item1\", \"item2\", \"item3\", \"item4\"}\n",
    "st2 = {\"item2\", \"item3\"}\n",
    "st2.symmetric_difference(st1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Truthy and falsy values\n",
    "\n",
    "Python objects can be used in expressions that will return a boolean value, such as when a list, `listy`, is used with `if listy`. Built-in Python objects that are empty are usually evaluated as `False`, and are said to be 'Falsy'. In contrast, when these built-in objects are not empty, they evaluate as `True` and are said to be 'truthy'.\n",
    "\n",
    "(If you are building your own classes, you can define this behaviour for them through the `__bool__` dunder method.)\n",
    "\n",
    "Let's see some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bool_check_var(input_variable):\n",
    "    if not (input_variable):\n",
    "        print(\"Falsy\")\n",
    "    else:\n",
    "        print(\"Truthy\")\n",
    "\n",
    "\n",
    "listy = []\n",
    "other_listy = [1, 2, 3]\n",
    "\n",
    "\n",
    "bool_check_var(listy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bool_check_var(other_listy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method we defined doesn't just operate on lists; it'll work for many various other truthy and falsy objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bool_check_var(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bool_check_var([0, 0, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that zero was falsy, its the nothing of a float, but a list of three zeros is not an empty list, so it evaluates as truthy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bool_check_var({})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bool_check_var(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing what is truthy or falsy is useful in practice; imagine you'd like to default to a specific behaviour if a list called `list_vals` doesn't have any values in. You now know you can do it simply with `if list_vals`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lambda functions\n",
    "\n",
    "Lambda functions are a very old idea in programming, and are part of the functional programming paradigm. Coding languages tend to be more object-oriented or functional, with the object-oriented approach originating with Alan Turing's \"Turing Machines\" and the functional approach with Alonso Church's \"lambda calculus\". These two approaches are mathematically equivalent and, on a more practical note, high-level programming languages often mix both. As examples, Haskell is strongly a functional language, statistics language R leans toward being more functional, Python is slightly more object oriented, and powerhouse languages like Fortran and C are object-oriented. However, despite being less functional than some languages, Python does have lambda functions, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1243e4e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "plus_one = lambda x: x + 1\n",
    "plus_one(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "517815ae",
   "metadata": {},
   "source": [
    "For a one-liner function that has a name it's actually better practice here to use `def plus_one(x): return x + 1`, so you shouldn't see this form of lambda function too much in the wild. However, you are likely to see lambda functions being used with dataframes and other objects. For example, if you had a dataframe with a column of string called 'strings' that you want to change to “Title Case” and replace one phrase with another, you could use lambda functions to do that (there are better ways of doing this but this is useful as a simple example):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b48977c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(\n",
    "    data=[[\"hello my blah is Ada\"], [\"hElLo mY blah IS Adam\"]],\n",
    "    columns=[\"strings\"],\n",
    "    dtype=\"string\",\n",
    ")\n",
    "df[\"strings\"].apply(lambda x: x.title().replace(\"Blah\", \"Name\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a8a27ab",
   "metadata": {},
   "source": [
    "More complex lambda functions can be constructed, eg `lambda x, y, z: x + y + z`. One of the best use cases of lambdas is when you *don't* want to go to the trouble of declaring a function. For example, let's say you want to compose a series of functions and you want to specify those functions in a list, one after the other. Using functions alone, you'd have to define a new function for each operation. With lambdas, it would look like this (again, there are easier ways to do this operation, but we'll use simple functions to demonstrate the principle):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "404c29ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "number = 1\n",
    "for func in [lambda x: x + 1, lambda x: x * 2, lambda x: x ** 2]:\n",
    "    number = func(number)\n",
    "    print(number)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdb2c40d",
   "metadata": {},
   "source": [
    "Note that people often use `x` by convention, but there's nothing to stop you writing `lambda horses: horses**2` (apart from the looks your co-authors will give you).\n",
    "\n",
    "```{admonition} Exercise\n",
    "Write a lambda function that takes the square root of an input number.\n",
    "```\n",
    "\n",
    "If you want to learn more about lambda functions, check out these [short video tutorials](https://calmcode.io/lambda/introduction.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splat and splatty-splat\n",
    "\n",
    "You read those right, yes. These are also known as \"unpacking operators\" for iterables that are fed into functions as arguments (in the form of a tuple) and keyword arguments (in the form of a dictionary) respectively. Splat is `*` and splatty-splat is `**`. Because they unpack, they allow us to efficiency send packages of arguments or keyword arguments into functions without labouriously writing out every single argument.\n",
    "\n",
    "Because function arguments are always tuples, the use of `*` must be accompanied by a tuple. Because function keywords are always dictionaries of key, value pairs, the use of `**` must always be accompanied by a dictionary.\n",
    "\n",
    "Let's take a look at splat, which unpacks tuples into function arguments. If we have a function that takes two arguments we can send variables to it in different ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89a10ac8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def add(a, b):\n",
    "    return a + b\n",
    "\n",
    "\n",
    "print(add(5, 10))\n",
    "\n",
    "func_args = (6, 11)\n",
    "\n",
    "print(add(*func_args))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ff21e6",
   "metadata": {},
   "source": [
    "The splat operator, `*`, unpacks the variable `func_args` into two different function arguments.\n",
    "\n",
    "Perhaps surprisingly, we can use the splat operator *in the definition of a function*. For example, sum_elements below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ef584b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sum_elements(*elements):\n",
    "    return sum(*elements)\n",
    "\n",
    "\n",
    "nums = (1, 2, 3)\n",
    "\n",
    "print(sum_elements(nums))\n",
    "\n",
    "more_nums = (1, 2, 3, 4, 5)\n",
    "\n",
    "print(sum_elements(more_nums))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1bcb685",
   "metadata": {},
   "source": [
    "```{admonition} Exercise\n",
    "Write a function multiply that multiplies two input numbers, `a` and `b`, together and returns the answer. Send the argument `(10, 12)` to it using the splat operator.\n",
    "```\n",
    "\n",
    "Splatty-splat, `**`, unpacks dictionaries into keyword arguments (aka kwargs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcd875f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def function_with_kwargs(a, x=0, y=0, z=0):\n",
    "    return a + x + y + z\n",
    "\n",
    "\n",
    "print(function_with_kwargs(5))\n",
    "\n",
    "kwargs = {\"x\": 3, \"y\": 4, \"z\": 5}\n",
    "\n",
    "print(function_with_kwargs(5, **kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83f29df",
   "metadata": {},
   "source": [
    "```{admonition} Exercise\n",
    "Using a dictionary and splatty-splat with the `function_with_kwargs` function, find the sum of 9, 6, 13, and 2.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Higher order functions\n",
    "\n",
    "Functions are like any other variable in Python, which means you can do some interesting things with them and, well, it can get a bit *meta*. For example, a function can take one or more functions as parameters, a function can be returned as a result of another function, functions can be defined within functions, a function can be assigned to a variable, and you can iterate over functions (for example, if they are in a list).\n",
    "\n",
    "Here's an example that shows how to use a higher order function: it accepts a function, `f`, as an argument and then, using the splat operator `*`, it accepts all arguments of that function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def join_a_string(str_list):\n",
    "    return \" \".join(str_list)\n",
    "\n",
    "\n",
    "def higher_order_function(f, *args):\n",
    "    \"\"\"Lowers case of result\"\"\"\n",
    "    out_string = f(*args)\n",
    "    return out_string.lower()\n",
    "\n",
    "\n",
    "result = higher_order_function(join_a_string, [\"Hello\", \"World!\"])\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next example, we show how to return a function from a function (assigning a function, `result`, to a variable in the process):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def square(x):\n",
    "    return x ** 2\n",
    "\n",
    "\n",
    "def cube(x):\n",
    "    return x ** 3\n",
    "\n",
    "\n",
    "def higher_order_function(type):  # a higher order function returning a function\n",
    "    if type == \"square\":\n",
    "        return square\n",
    "    elif type == \"cube\":\n",
    "        return cube\n",
    "\n",
    "\n",
    "result = higher_order_function(\"square\")\n",
    "print(f\"Using higher_order_function('square'), result(3) yields {result(3)}\")\n",
    "result = higher_order_function(\"cube\")\n",
    "print(f\"Using higher_order_function('cube'), result(3) yields {result(3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Functions within functions are allowed. They are known as *closures*. Here's a simple (if contrived) example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "\n",
    "def print_time_now():\n",
    "    def get_curr_time():\n",
    "        return datetime.now().strftime(\"%H:%M\")\n",
    "\n",
    "    now = get_curr_time()\n",
    "    print(now)\n",
    "\n",
    "\n",
    "print_time_now()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's see how to iterate over functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def square_root(x):\n",
    "    return x ** (0.5)\n",
    "\n",
    "\n",
    "functions_list = [square_root, square, cube]\n",
    "\n",
    "for func in functions_list:\n",
    "    print(f\"{func.__name__} applied to 4 is {func(4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterators\n",
    "\n",
    "An iterator is an object that contains a countable number of values that a single command, `next`, iterates through. Before that's possible though, we need to take a countable group of some kind and use the `iter` keyword on it to turn it into an iterator. Let's see an example with some text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_lst = [\"Mumbai\", \"Delhi\", \"Bangalore\"]\n",
    "\n",
    "myiterator = iter(text_lst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, nothing has happened yet, but that's because we didn't call it yet. To get the next iteration, whatever it is, use `next`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "next(myiterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "next(myiterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "next(myiterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, we've been through all of the values so... what's going to happen `next`!?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "next(myiterator)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "---------------------------------------------------------------------------\n",
    "StopIteration                             Traceback (most recent call last)\n",
    "<ipython-input-27-29fb3b4dbbec> in <module>\n",
    "----> 1 next(myiterator)\n",
    "\n",
    "StopIteration: \n",
    "```\n",
    "\n",
    "Iterating beyond the end raises a `StopIteration` error because we reached the end. To keep going, use `cycle` in place of `iter`. Note that you can build your own iterators (here we used a built-in object type, the `list`, to create an iterator of type `list_iterator`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generators\n",
    "\n",
    "Generator functions return 'lazy' iterators. They are lazy because they do not store their contents in memory. This has *big* advantages for some operations in specific situations: datasets larger than can fit into your computer's memory, or a complex function that needs to maintain an internal state every time it’s called.\n",
    "\n",
    "To give an idea of how and when they work, imagine that (exogeneously) integers are really costly, taking as much as 10 MB of space to store (the real figure is more like 128 bytes). We will write a function, \"firstn\", that represents the first $n$ non-negative integers, where $n$ is large. The most naive possible way of doing this would be to build the full list in memory like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def first_n_naive(n):\n",
    "    \"\"\"Build and return a list\"\"\"\n",
    "    num, nums = 0, []\n",
    "    while num < n:\n",
    "        nums.append(num)\n",
    "        num += 1\n",
    "    return nums\n",
    "\n",
    "\n",
    "sum_of_first_n = sum(first_n_naive(1000000))\n",
    "sum_of_first_n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `nums` stores *every* number before returning all of them. In our imagined case, this is completely infeasible because we don't have enough computer space to keep all $n$ 10MB integers in memory.\n",
    "\n",
    "Now we'll rewrite the list-based function as a generator-based function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def first_n_generator(n):\n",
    "    \"\"\"A generator that yields items instead of returning a list\"\"\"\n",
    "    num = 0\n",
    "    while num < n:\n",
    "        yield num\n",
    "        num += 1\n",
    "\n",
    "\n",
    "sum_of_first_n = sum(first_n_generator(1000000))\n",
    "sum_of_first_n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, instead of creating an enormous list that has to be stored in memory, we `yield` up each number as it is 'generated'. The cleverness that's going on here is that  the 'state' of the function is remembered from one call to the next. This means that when `next` is called on a generator object (either explicitly or implicitly, as in this example), the previously yielded variable `num` is incremented, and then yielded again.\n",
    "\n",
    "That was a fairly contrived example but there are plenty of practical ones. Working with pipelines that process very large datasets is a classic use case. For example, imagine you have a csv file that's far too big to fit in memory, i.e. open all at once, but you'd like to check the contents of each row and perhaps process them. The code below would `yield` each row in turn.\n",
    "\n",
    "```python\n",
    "def csv_reader(file_name):\n",
    "    for row in open(file_name, \"r\"):\n",
    "        yield row\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An even more concise way of defining this is via a *generator expression*, which syntactically looks a lot like a *list comprehension* but is a generator rather than a list. The example we just saw would be written as:\n",
    "\n",
    "```python\n",
    "csv_gen = (row for row in open(file_name))\n",
    "```\n",
    "\n",
    "It's easier to see the difference in the below example which clearly shows the analogy between *list comprehensions* and *generator comprehensions*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sq_nums_lc = [num ** 2 for num in range(2, 6)]\n",
    "sq_nums_lc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sq_nums_gc = (num ** 2 for num in range(2, 6))\n",
    "sq_nums_gc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The latter is a generator object and we can only access individual values calling `next` on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "next(sq_nums_gc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for small numbers of entries, lists may actually be faster and more efficient than generators-but for large numbers of entries, generators will almost always win out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decorators\n",
    "\n",
    "Decorators 'decorate' functions, they adorn them, modifying them as they execute. Let's say we want to run some numerical functions but we'd like to add ten on to whatever results we get. We could do it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiply(num_one, num_two):\n",
    "    return num_one * num_two\n",
    "\n",
    "\n",
    "def add_ten(in_num):\n",
    "    return in_num + 10\n",
    "\n",
    "\n",
    "answer = add_ten(multiply(3, 4))\n",
    "answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is fine for a one-off but a bit tedious if we're going to be using `add_ten` a lot, and on many functions. Decorators allow for a more general solution that can be applied, in this case, to any `inner` function that has two arguments and returns a numeric value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_ten(func):\n",
    "    def inner(a, b):\n",
    "        return func(a, b) + 10\n",
    "\n",
    "    return inner\n",
    "\n",
    "\n",
    "@add_ten\n",
    "def multiply(num_one, num_two):\n",
    "    return num_one * num_two\n",
    "\n",
    "\n",
    "multiply(3, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the same decorator for a different function (albeit one of the same form) now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@add_ten\n",
    "def divide(num_one, num_two):\n",
    "    return num_one / num_two\n",
    "\n",
    "\n",
    "divide(10, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the magic of decorators is such that we can define them for much more general cases, regardless of the number of arguments or even keyword arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_ten(func):\n",
    "    def inner(*args, **kwargs):\n",
    "        print(\"Function has been decorated!\")\n",
    "        print(\"Adding ten...\")\n",
    "        return func(*args, **kwargs) + 10\n",
    "\n",
    "    return inner\n",
    "\n",
    "\n",
    "@add_ten\n",
    "def combine_three_nums(a, b, c):\n",
    "    return a * b - c\n",
    "\n",
    "\n",
    "@add_ten\n",
    "def combine_four_nums(a, b, c, d=0):\n",
    "    return a * b - c - d\n",
    "\n",
    "\n",
    "combine_three_nums(1, 2, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now see it applied to a function with a different number of (keyword) arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "combine_four_nums(3, 4, 2, d=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decorators can be chained too (and order matters):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dividing_line(func):\n",
    "    def inner(*args, **kwargs):\n",
    "        print(\"\".join([\"-\"] * 30))\n",
    "        out = func(*args, **kwargs)\n",
    "        return out\n",
    "\n",
    "    return inner\n",
    "\n",
    "\n",
    "@dividing_line\n",
    "@add_ten\n",
    "def multiply(num_one, num_two):\n",
    "    return num_one * num_two\n",
    "\n",
    "\n",
    "multiply(3, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Errors and exceptions\n",
    "\n",
    "When a programme goes wrong, it throws up an error and halts. You won't be coding for long before you hit one of these errors, which have special names depending on what triggered them.\n",
    "\n",
    "Let's see a real-life error in action:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "denom = 0\n",
    "\n",
    "print(1/denom)\n",
    "```\n",
    "\n",
    "```python\n",
    "---------------------------------------------------------------------------\n",
    "ZeroDivisionError                         Traceback (most recent call last)\n",
    "<ipython-input-39-e45c0e0a3e37> in <module>\n",
    "      1 denom = 0\n",
    "      2 \n",
    "----> 3 print(1/denom)\n",
    "\n",
    "ZeroDivisionError: division by zero\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no! We got a `ZeroDivisionError` and our programme crashed. Note that the error includes a 'Traceback' to show which line went wrong, which is helpful for debugging.\n",
    "\n",
    "In practice, there are often times when we know that an error *could* arise, and we would like to specify what should happen when it does (rather than having the programme crash). \n",
    "\n",
    "We can use *exceptions* to do this. These come in a `try` ... `except` pattern, which looks like an `if` ... `else` pattern but applies to errors. If no errors occur inside the `try` block, the `except` block isn’t run but *if* something goes wrong inside the `try` then the `except` block is executed. Let's see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for denom in [-5, 0, 5]:\n",
    "    try:\n",
    "        result = 1 / denom\n",
    "        print(f\"1/{denom} == {result}\")\n",
    "    except:\n",
    "        print(f\"Cannot divide by {denom}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see two differences. First: the code executed just fine *without* halting. Second: when we hit the error, the `except` block was executed and told us what was going on.\n",
    "\n",
    "In this case, we wrote an informative message about the error but it's convenient to use Python's built in messages where we can. In the below, not only do we send our own message about the error but we add info on what caused the error for the language too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for denom in [-5, 0, 5]:\n",
    "    try:\n",
    "        result = 1 / denom\n",
    "        print(f\"1/{denom} == {result}\")\n",
    "    except Exception as error:\n",
    "        print(f\"{denom} has no reciprocal; error is: {error}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sadly, division by zero is just one of the many errors you might encounter. What if a function is likely to end up running into several different errors? We can have multiple `except` clauses to catch these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = [-5, 0, 5]\n",
    "for i in [0, 1, 2, 3]:\n",
    "    try:\n",
    "        denom = numbers[i]\n",
    "        result = 1 / denom\n",
    "        print(f\"1/{denom} == {result}\")\n",
    "    except IndexError as error:\n",
    "        print(f\"index {i} out of range; error is {error}\")\n",
    "    except ZeroDivisionError as error:\n",
    "        print(f\"{denom} has no reciprocal; error is: {error}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A full list of built-in errors may be [found here](https://docs.python.org/3/library/exceptions.html#exception-hierarchy) and they are nested in classes (eg `ZeroDivisionError` is a special case of a `ArithmeticError`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where do these errors come from anyway? What tells the programming language to throw a tantrum when it encounters certain combinations of values and operations.\n",
    "\n",
    "The answer is that the person or people who wrote the code that's 'under the hood' can specify when such errors should be raised. Remember, the philosophy of Python is that things should faily loudly (so that they do not cause issues downstream). Here's an example of some code that raises its own errors using the `raise` keyword:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for number in [1, 0, -1]:\n",
    "    try:\n",
    "        if number < 0:\n",
    "            raise ValueError(f\"no negatives: {number}\")\n",
    "        print(number)\n",
    "    except ValueError as error:\n",
    "        print(f\"exception: {error}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `ValueError` is a built-in type of error and there are plenty of ones to choose from for your case. Some big or specialised libraries define their own types of error too.\n",
    "\n",
    "One very clever feature of Python's exception handling is \"throw low, catch high\", which means that even if an error gets thrown way deep down in the middle of a code block, the catching exception can be used some way away. Here's an example: the error arises *within* the `sum_reciprocals` function, but is caught elsewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sum_reciprocals(values):\n",
    "    result = 0\n",
    "    for v in values:\n",
    "        result += 1 / v\n",
    "    return result\n",
    "\n",
    "\n",
    "numbers = [-1, 0, 1]\n",
    "try:\n",
    "    one_over = sum_reciprocals(numbers)\n",
    "except ArithmeticError as error:\n",
    "    print(f\"Error trying to sum reciprocals: {error}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example of combining a `try`, several `except` statements, an `else` that gets executed if `try` is, and a `finally` that always get executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "b_year = 1852\n",
    "name_input = \"Ada\"\n",
    "\n",
    "\n",
    "def process_input(name_input, b_year):\n",
    "    try:\n",
    "        name = str(name_input)\n",
    "        year_born = int(b_year)\n",
    "        age = datetime.datetime.now().year - int(year_born)\n",
    "        print(f\"You are {name}. And your age is {age}.\")\n",
    "    except TypeError:\n",
    "        print(\"Type error occur\")\n",
    "    except ValueError:\n",
    "        print(\"Value error occur\")\n",
    "    except ZeroDivisionError:\n",
    "        print(\"zero division error occur\")\n",
    "    else:\n",
    "        print(\"I usually run with the try block\")\n",
    "    finally:\n",
    "        print(\"I always run.\")\n",
    "\n",
    "\n",
    "process_input(name_input, b_year)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classes and objects\n",
    "\n",
    "Python is an object oriented programming language. Everything is an object (and every object has a type). A Class is an object constructor, a blueprint for creating objects. An object is a 'live' instance of a class. Objects are to classes what a yellow VW Beetle is to cars. The class defines the attributes and methods that the object can perform.\n",
    "\n",
    "Classes and instances of them are useful in certain situations, the most common being when you need something that has 'state', i.e. it can remember things that have happened to it, carry information with it, and change form.\n",
    "\n",
    "While you're quite unlikely to need to build classes in economics (unless you're doing something really fancy), some of the biggest Python packages are based around classes so it's useful to understand a bit about how they work, and especially how they have state.\n",
    "\n",
    "The syntax to create a class is\n",
    "\n",
    "```python\n",
    "class ClassName:\n",
    "  ...code...\n",
    "```\n",
    "\n",
    "But it's easiest to show with an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a class called Person\n",
    "\n",
    "\n",
    "class Person:\n",
    "    def __init__(self, name):\n",
    "        self.name = name\n",
    "\n",
    "\n",
    "# Create an instance of the class\n",
    "p = Person(\"Adam\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we check `type`, that's when it gets *really* interesting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah! We created a whole new data type based on the `Class` name. The class has a constructor method, `__init__`, that, in this case, takes an input variable `name` and assigns it to an *internal* object variable `name`. The `self` variable that you can also see is really saying 'generate an object of type this Class when called'. We can access any internal variables like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay but what's the point of all this? Well we can now create as many objects as we like of class 'Person' and they will have the same structure, but not the same state, as other objects of class 'Person'.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = Person(\"Ada\")\n",
    "m.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a very boring class! Let's add a method, which will allow us to change the state of objects. Here, we add a method `increment_age` which is also indented under the `class Person` header. Note that it takes `self` as an input, just like the constructor, but it only acts on objects of type person that have *already* been created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Person:\n",
    "    def __init__(self, name, age):\n",
    "        self.name = name\n",
    "        self.age = age\n",
    "\n",
    "    def increment_age(self):\n",
    "        self.age = self.age + 1\n",
    "\n",
    "\n",
    "# Create an instance of the class\n",
    "p = Person(\"Adam\", 231)\n",
    "\n",
    "print(p.age)\n",
    "# Call the method increment_age\n",
    "p.increment_age()\n",
    "print(p.age)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This very simple method changes the internal state. Just like class constructors and regular functions, class methods can take arguments. Here's an example:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Person:\n",
    "    def __init__(self, name, age):\n",
    "        self.name = name\n",
    "        self.age = age\n",
    "\n",
    "    def increment_age(self):\n",
    "        self.age = self.age + 1\n",
    "\n",
    "    def change_age(self, new_age):\n",
    "        self.age = new_age\n",
    "\n",
    "\n",
    "# Create an instance of the class\n",
    "p = Person(\"Adam\", 231)\n",
    "\n",
    "print(p.age)\n",
    "# Call the method increment_age\n",
    "p.change_age(67)\n",
    "print(p.age)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be tedious to have to initialise a class with a whole load of parameters every time. Just like with functions, we can define *default parameters* for classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Person:\n",
    "    def __init__(self, name=\"default_name\", age=20):\n",
    "        self.name = name\n",
    "        self.age = age\n",
    "\n",
    "\n",
    "p = Person()\n",
    "p.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That covers a lot of the basics of classes but if you're using classes in anger then you might also want to look up [inheritance and composition](https://realpython.com/inheritance-composition-python/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataclassess\n",
    "\n",
    "The basic classes we created above come with a lot of 'boilerplate'; code we need but which is not very surprising. Dataclasses were inroduced in Python 3.7 as a way to remove this boilerplate when the classes being created are quite simple. Think of dataclasses as a class with sensible defaults that is for light object-oriented programming.\n",
    "\n",
    "A simple example, with a `Circle` class, demonstrates why they are effective. First, the full class way of doing things:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "class Circle1:\n",
    "    def __init__(self, colour: str, radius: float) -> None:\n",
    "        self.colour = colour\n",
    "        self.radius = radius\n",
    "\n",
    "    def area(self) -> float:\n",
    "        return np.pi * self.radius ** 2\n",
    "\n",
    "\n",
    "circle1 = Circle1(\"red\", 2)\n",
    "circle1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't get a very informative message when we call `circle1`, as you can see. At least we can compute its area:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "circle1.area()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll create the same object with dataclasses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataclasses import dataclass\n",
    "\n",
    "\n",
    "@dataclass\n",
    "class Circle2:\n",
    "    colour: str\n",
    "    radius: float\n",
    "\n",
    "    def area(self) -> float:\n",
    "        return np.pi * self.radius ** 2\n",
    "\n",
    "\n",
    "circle2 = Circle2(\"blue\", 2)\n",
    "circle2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right away we get a much more informative message when we call the object, *and* the class definition is a whole lot simpler. Everything else is just the same (just try calling `circle2.area()`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tests\n",
    "\n",
    "Tests check that your code is behaving as you expect, even as parts of it change or are updated. They most commonly compare an expected input and output with what actually comes out of your code for a given input.\n",
    "\n",
    "Writing tests for code is often an after thought for research code, if it's a thought at all. That's understandable. But it can really boost reliability and robustness. This excerpt from [Research Software Engineering with Python](https://merely-useful.github.io/py-rse/testing.html) explains why testing even research code is extremely useful:\n",
    "\n",
    "> Why is testing research software important? A successful early career researcher in protein crystallography, Geoffrey Chang, had to retract five published papers—three from the journal Science—because his code had inadvertently flipped two columns of data (Miller 2006). More recently, a simple calculation mistake in a paper by Reinhart and Rogoff contributed to making the financial crash of 2008 even worse for millions of people (Borwein and Bailey 2013). Testing helps to catch errors like these.\n",
    "\n",
    "The Reinhart and Rogoff paper used Excel for its analysis; while Excel has some pros and does some good in the world, my strong recommendation is that it should *not* be used for any important analysis.\n",
    "\n",
    "Back to testing: if you're writing a *package* of code for others to use, one that will change and grow over time, tests are essential. These are some general guidelines for good testing, such as:\n",
    "\n",
    "- A testing 'unit' should focus on a single bit of functionality;\n",
    "- Each test unit must be fully independent (in practice, this means that `setUp()` and `tearDown()` methods need to be defined to prep data or objects to the point where they can be used by the testing unit);\n",
    "- Tests should run fast if possible;\n",
    "- If you know how, it's a good idea to use something like a 'Git hook' that runs tests before code is saved in a shared repository;\n",
    "- If you find a bug in your code, it's good practice to write a new test that targets it; and\n",
    "- Although long, extremely descriptive names are not very helpful in regular code, they *should* be used for functions that test code because this is the name you will see when the test fails (and you want to know what it refers to).\n",
    "\n",
    "The most common way to run tests is to add assertions to code. An assertion is a statement that something must be true at a certain point in a program. When an assertion evaluates to true, nothing happens; if it’s false, the programme stops and prints a user-defined error message. Here's an example of an assertion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_num = 5\n",
    "\n",
    "assert positive_num > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we had run \n",
    "\n",
    "```python\n",
    "positive_num = 5\n",
    "\n",
    "assert positive_num < 0\n",
    "```\n",
    "\n",
    "it would have resulted in\n",
    "\n",
    "```python\n",
    "---------------------------------------------------------------------------\n",
    "AssertionError                            Traceback (most recent call last)\n",
    "<ipython-input-55-db0e55fa5cb7> in <module>\n",
    "      1 positive_num = 5\n",
    "      2 \n",
    "----> 3 assert positive_num < 0\n",
    "\n",
    "AssertionError:\n",
    "``` \n",
    "\n",
    "Putting assertions into code here and there is one way to check that everything is working as you anticipate, but you won't find out about it until you actually run the code! A testing framework allows you to separate checking the code behaves as you'd expect from running the code, so that you can ensure everything works before putting that big run on.\n",
    "\n",
    "To run lots of tests, we can use a test framework (also called a test runner). A commonly used test framework is [**pytest**](https://docs.pytest.org/en/latest/). Essentially it will run anything that you've flagged as a test in your code and report back to you on whether it passed or not. Here's how it works: tests are put in files whose names begin with `test_`, each test is a function whose name also begins with `test_`, and these functions use `assert` statements to check results.\n",
    "\n",
    "Normally, your functions would be in a different script but here's an example of a code function and a testing function (that tests it) that you might put in a file called `test.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# content of test.py\n",
    "def inc(x):\n",
    "    return x + 1\n",
    "\n",
    "\n",
    "def test_answer():\n",
    "    assert inc(3) == 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run this, we would enter\n",
    "\n",
    "```bash\n",
    "pytest\n",
    "```\n",
    "\n",
    "on the command line. This would yield\n",
    "\n",
    "```bash\n",
    "=========================== test session starts ============================\n",
    "platform linux -- Python 3.x.y, pytest-6.x.y, py-1.x.y, pluggy-0.x.y\n",
    "cachedir: $PYTHON_PREFIX/.pytest_cache\n",
    "rootdir: $REGENDOC_TMPDIR\n",
    "collected 1 item\n",
    "\n",
    "test.py F                                                     [100%]\n",
    "\n",
    "================================= FAILURES =================================\n",
    "_______________________________ test_answer ________________________________\n",
    "\n",
    "    def test_answer():\n",
    ">       assert inc(3) == 5\n",
    "E       assert 4 == 5\n",
    "E        +  where 4 = inc(3)\n",
    "\n",
    "test_sample.py:6: AssertionError\n",
    "========================= short test summary info ==========================\n",
    "FAILED test.py::test_answer - assert 4 == 5\n",
    "============================ 1 failed in 0.12s =============================\n",
    "```\n",
    "\n",
    "As you add more tests, they will be automatically picked up and run by `pytest`.\n",
    "\n",
    "When writing tests, think about whether you have tested realistic combinations of input parameters, tested all discrete outputs at least once, tested the boundaries of continuous outputs, and ensured that informative errors are raised when things go wrong.\n",
    "\n",
    "If you're using Visual Studio Code, support for **pytest** comes built-in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing continuous variables\n",
    "\n",
    "Imagine we are not testing for an integer, such as 4, but instead for a real number, such as 4.32838. Computers are nothing if not punishingly literal, however, so if evaluation of the code produces 4.32837 instead of 4.32838 the `assert` statement will fail and the test won't pass even if the two numbers are close enough for our purposes. It's even worse than that because computers *cannot* accurately represent real numbers to arbitrary levels of precision. \n",
    "\n",
    "To avoid tests that fail even though nothing's really wrong, most testing packages come with tools for real numbers. Here's an example where we have a parameter alpha that we'd like to test against an expected value of 1 but with a tolerance of 0.01."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pytest\n",
    "\n",
    "\n",
    "def complicated_func():\n",
    "    \"\"\"A really complicated func\"\"\"\n",
    "    return 0.998\n",
    "\n",
    "\n",
    "alpha = complicated_func()\n",
    "expected_alpha = pytest.approx(1.0, abs=0.01)\n",
    "assert alpha == expected_alpha\n",
    "print(expected_alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test coverage\n",
    "\n",
    "A production-ready code should be heavily tested. Coverage tests ask how much of your code is actually covered by tests and where the code is that isn't tested. One package to do this is called [**coverage**](https://coverage.readthedocs.io/en/coverage-5.5/). If you already have code tests written, you can run:\n",
    "\n",
    "```bash\n",
    "coverage run -m pytest\n",
    "```\n",
    "\n",
    "on the command line instead of just running `pytest`. To get a report on the success use\n",
    "\n",
    "```bash\n",
    "$ coverage report -m\n",
    "Name                      Stmts   Miss  Cover   Missing\n",
    "-------------------------------------------------------\n",
    "my_program.py                20      4    80%   33-35, 39\n",
    "my_other_module.py           56      6    89%   17-23\n",
    "-------------------------------------------------------\n",
    "TOTAL                        76     10    87%\n",
    "```\n",
    "\n",
    "If you would prefer a html report over one in the terminal, use `$ coverage html` to create a report at `htmlcov/index.html`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Type annotations and type checkers\n",
    "\n",
    "Type annotations were introduced in Python 3.5 (these notes are written in 3.8). If you've seen more low-level languages, typing will be familiar to you. Python uses 'duck typing' (\"if it walks and quacks like a duck, it is a duck\") which means that if a variable walks like an integer, and talks like an integer, then it gets treated as if it is an integer. Ditto for other variable types. Duck typing is useful if you just want to code quickly and aren't writing production code.\n",
    "\n",
    "But... there are times when you *do* know what variable types you're going to be dealing with ahead of time and you want to prevent the propagation of the wrong kinds of variable types. In these situations, you can clearly say what variable types are supposed to be. And, when used with some other packages, typing can make code easier to understand, debug, and maintain.\n",
    "\n",
    "Note that it doesn't have to be all or nothing on type checking, you can just add it in gradually or where you think it's most important.\n",
    "\n",
    "Now it's important to be really clear on one point, namely that *Python does **not** enforce type annotations*. But we can use *static type checking* to ensure all types are as they should be in *advance* of running. Before we do that, let's see how we add type annotations. \n",
    "\n",
    "This is the simplest example of a type annotation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer: int = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This explicitly says that `answer` is an integer. Type annotations can be used in functions too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def increment(number: int) -> int:\n",
    "    return number + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A static type checker uses these type annotations to verify the type correctness of a programme without executing it. [**mypy**](http://mypy-lang.org/) is the most widely used static type checker. After installing **mypy**, to run type checking on a file `code_script.py` use\n",
    "\n",
    "```bash\n",
    "mypy code_script.py\n",
    "```\n",
    "\n",
    "on the command line.\n",
    "\n",
    "What do you see when you run it? Let's say the content of your script is:\n",
    "\n",
    "```python\n",
    "# Contents of code_script.py\n",
    "def greeting(name: str) -> str:\n",
    "    return 'Hello ' + name\n",
    "\n",
    "\n",
    "greeting(3)\n",
    "```\n",
    "\n",
    "This would return:\n",
    "\n",
    "```bash\n",
    "Argument 1 to \"greeting\" has incompatible type \"int\"; expected \"str\"\n",
    "```\n",
    "\n",
    "Here are more of the type annotations that you might need or come across, courtesy of the [**mypy**](http://mypy-lang.org/) documentation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from typing import List, Set, Dict, Tuple, Optional\n",
    "\n",
    "# For simple built-in types, just use the name of the type\n",
    "x: int = 1\n",
    "x: float = 1.0\n",
    "x: bool = True\n",
    "x: str = \"test\"\n",
    "x: bytes = b\"test\"\n",
    "\n",
    "# For collections, the type of the collection item is in brackets\n",
    "# (Python 3.9+ only)\n",
    "x: list[int] = [1]\n",
    "x: set[int] = {6, 7}\n",
    "\n",
    "# In Python 3.8 and earlier, the name of the collection type is\n",
    "# capitalized, and the type is imported from 'typing'\n",
    "x: List[int] = [1]\n",
    "x: Set[int] = {6, 7}\n",
    "\n",
    "# Same as above, but with type comment syntax (Python 3.5 and earlier)\n",
    "x = [1]  # type: List[int]\n",
    "\n",
    "# For mappings, we need the types of both keys and values\n",
    "x: dict[str, float] = {'field': 2.0}  # Python 3.9+\n",
    "x: Dict[str, float] = {'field': 2.0}\n",
    "\n",
    "# For tuples of fixed size, we specify the types of all the elements\n",
    "x: tuple[int, str, float] = (3, \"yes\", 7.5)  # Python 3.9+\n",
    "x: Tuple[int, str, float] = (3, \"yes\", 7.5)\n",
    "\n",
    "# For tuples of variable size, we use one type and ellipsis\n",
    "x: tuple[int, ...] = (1, 2, 3)  # Python 3.9+\n",
    "x: Tuple[int, ...] = (1, 2, 3)\n",
    "\n",
    "# Use Optional[] for values that could be None\n",
    "x: Optional[str] = some_function()\n",
    "# Mypy understands a value can't be None in an if-statement\n",
    "if x is not None:\n",
    "    print(x.upper())\n",
    "# If a value can never be None due to some invariants, use an assert\n",
    "assert x is not None\n",
    "print(x.upper())\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Timing and profiling\n",
    "\n",
    "Timing and profiling your code are useful to understand where bottlenecks might be. One of the principles of programming is don't optimise too soon, so, if you are reaching for timing and profiling, you've probably already hit that bottleneck.\n",
    "\n",
    "```{note}\n",
    "This section just deals with diagnosing slow code: how to speed code up will be covered elsewhere.\n",
    "```\n",
    "\n",
    "There are different levels of sophistication for timing and profiling. The simplest way to time is to use the built-in library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import timeit\n",
    "\n",
    "\n",
    "def f(nsec=1.0):\n",
    "    \"\"\"Function sleeps for nsec seconds.\"\"\"\n",
    "    time.sleep(nsec)\n",
    "\n",
    "\n",
    "start = timeit.default_timer()\n",
    "f()\n",
    "elapsed = timeit.default_timer() - start\n",
    "elapsed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're working jupyter notebooks (which is what this book is written in), you can use the `timeit` magic, a convenient shortcut for timing a line or chunk of code (more on magics, and timeit, [here](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-time)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit f(0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "timeit will adjust the number of repeats according to how slow the code is to run. Several other magics are available:\n",
    "\n",
    "- `%time`: Time the execution of a single statement\n",
    "- `%timeit`: Time repeated execution of a single statement for more accuracy\n",
    "- `%prun`: Run code with the profiler\n",
    "- `%lprun`: Run code with the line-by-line profiler\n",
    "- `%memit`: Measure the memory use of a single statement\n",
    "- `%mprun`: Run code with the line-by-line memory profiler\n",
    "\n",
    "Another way to time functions is via decorators:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_time(f, *args, **kwargs):\n",
    "    def func(*args, **kwargs):\n",
    "        import timeit\n",
    "\n",
    "        start = timeit.default_timer()\n",
    "        f(*args, **kwargs)\n",
    "        print(timeit.default_timer() - start)\n",
    "\n",
    "    return func\n",
    "\n",
    "\n",
    "@process_time\n",
    "def f1(nsec=1.0):\n",
    "    \"\"\"Function sleeps for nsec seconds.\"\"\"\n",
    "    time.sleep(nsec)\n",
    "\n",
    "\n",
    "f1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Timing is very simple and just tells you the total time of what's inbetween the start and end points. Often, you would want a bit more detail about what's being slow, or you may wish to profile an entire script. For that, there's `cProfile`, which is a built-in package. You can use it on the command line or within scripts, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import cProfile\n",
    "\n",
    "\n",
    "def foo():\n",
    "    time.sleep(1.0)\n",
    "    time.sleep(2.5)\n",
    "\n",
    "\n",
    "cProfile.run(\"foo()\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do we get here? The `percall` columns are `tottime` and `cumtime` (respectively) divided by `ncalls`. `tottime` is the time spent in the function excluding any sub-functions, while `cumtime` is the cumulative time spent.\n",
    "\n",
    "There are various profilers out there that extend the basics provided by **cProfile**. Let's look at a few others to get a sense of what extra they offer.\n",
    "\n",
    "[**pyinstrument**](https://github.com/joerick/pyinstrument) attempts to improve on the standard profilers by recording the 'entire stack' of methods so that identifying expensive sub-calls to methods is easier. There are two ways to run it: on the command line, using `pyinstrument script.py` instead of `python script.py`, or by wedging the code you're interested in *within* a script between a `.start()` and `.stop()` method like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyinstrument import Profiler\n",
    "\n",
    "\n",
    "profiler = Profiler()\n",
    "profiler.start()\n",
    "\n",
    "\n",
    "def fibonacci(n):\n",
    "    if n < 0:\n",
    "        raise Exception(\"n must be a positive integer\")\n",
    "    elif n == 1:\n",
    "        return 0\n",
    "    elif n == 2:\n",
    "        return 1\n",
    "    else:\n",
    "        return fibonacci(n - 1) + fibonacci(n - 2)\n",
    "\n",
    "\n",
    "fibonacci(20)\n",
    "\n",
    "profiler.stop()\n",
    "\n",
    "# Print the results of the profiling to screen\n",
    "print(profiler.output_text(unicode=True, color=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because our function, `fibonacci`, is nested, the function call breakdown that **pyinstrument** produces is likewise nested so that you can see how the code calls progresses.\n",
    "\n",
    "The output analysis can be written to an interactive HTML file too: just use `profiler.output_html()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to execute code is not the only thing we care about. In fact, for economics applications, it's much more likely to be a memory bottleneck than a speed one. Operations that involve lots of data, large matrices, or both, might be memory hogs. If you hit the memory (RAM) limit on your machine, it will slow right down and the process that's running might just crash completely. So wouldn't it be great it we had a way to profile both code and memory!? Well, we do.\n",
    "\n",
    "[**scalene**](https://github.com/plasma-umass/scalene) is a command-line utility for profiling code execution time *and* memory. It profiles whole scripts, run `scalene script.py` on the command line, and produces HTML reports. Here's an example of some output:\n",
    "\n",
    "![Screenshot from scalene running on a problem](https://raw.githubusercontent.com/plasma-umass/scalene/master/docs/images/sample-profile-pystone.png)\n",
    "\n",
    "What's particularly useful is having the part of the code where the bottleneck is displayed on the right. Memory usage over time tells you where the memory-hogging lines are, while the two CPU % columns tell you how much of the total time (shown at the top) was spent in Python code and in non-Python ('native') code, which Python often calls under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I am the Walrus\n",
    "\n",
    "The Walrus operator, `:=` was introduced in Python 3.8 and, well, it's fairly complicated but it *does* have its uses. The main use case for the Walrus operator is when you want to both *evaluate an expression* and *assign a variable* in one fell swoop.\n",
    "\n",
    "Take this (trivial) example which involves evaluating an expression, `len(a) > 4`, that returns a boolean and then assigning that *same* expression to a variable `n`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = [1, 2, 3, 4]\n",
    "if len(a) > 3:\n",
    "    n = len(a)\n",
    "    print(f\"List is too long ({n} elements, expected <= 3)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Walrus operator allows us to skip the clumsy use of `len(a)` twice and do both steps in one go. As noted, that's trivial here, but if evaluation were very computationally expensive, then this might save us some trouble. Here's the version with the Walrus operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = [1, 2, 3, 4]\n",
    "if (n := len(a)) > 3:\n",
    "    print(f\"List is too long ({n} elements, expected <= 3)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple dispatch\n",
    "\n",
    "One can use object-oriented methods and inheritance to get different code objects to behave in different ways depending on the type of input. For example, a different behaviour might occur if you send a string into a function versus an integer. An alternative to the object-oriented approach is to use *multiple dispatch*. [**fastcore**](https://fastcore.fast.ai/) is a library that provides \"goodies to make your coding faster, easier, and more maintainable\" and has many neat features but amongst the 'goodies' is multiple dispatch, with the `typedispatch` decorator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fastcore is designed to be imported as *\n",
    "from fastcore.dispatch import *\n",
    "\n",
    "\n",
    "@typedispatch\n",
    "def func_example(x: int, y: float):\n",
    "    return x + y\n",
    "\n",
    "\n",
    "@typedispatch\n",
    "def func_example(x: int, y: int):\n",
    "    return x * y\n",
    "\n",
    "\n",
    "# Int and float\n",
    "print(func_example(5, 5.0))\n",
    "\n",
    "# Int and int\n",
    "print(func_example(5, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we can see here is that we have the same function, `func_example`, used twice with *very* similar inputs. But the inputs are *not* the same; in the first instance it's an integer and a float while in the second it's two integers. The different inputs get routed into the different versions of the `@typeddispatch` function. This decorator-based approach is not the only way to use **fastcore** to do typed dispatch but it's one of the most convenient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map, filter, and reduce\n",
    "\n",
    "Map, filter, and reduce are built-in higher order functions. Lambda functions, featured in the basics of coding chapter, can be passed as into each of these as an argument and some of the best use cases of lambda functions are in conjunction with map, filter, and reduce.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map\n",
    "\n",
    "`map` takes a function and an iterable as arguments, ie the syntax is `map(function, iterable)`. An iterable is a type that is composed of elements that can be iterated over. The map essentially applies the function to each entry in the iterable. Here's an example where a list of strings is cast to integers via `map`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers_str = [\"1\", \"2\", \"3\", \"4\", \"5\"]\n",
    "mapped_result = map(int, numbers_str)\n",
    "list(mapped_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example with a lambda function. The benefit of using a lambda in this map operation is that otherwise we would have to write a whole function that simply returned the input with `.title()` at the end:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = [\"robinson\", \"fawcett\", \"ostrom\"]\n",
    "names_titled = map(lambda name: name.title(), names)\n",
    "list(names_titled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter\n",
    "\n",
    "`filter` calls a specified function and returns a boolean for each item of the specified iterable. It filters the items that satisfy the given boolean criteria. It uses the `filter(function, iterable)` syntax. In the example below, we take all the numbers from zero to five and filter them according to whether they are divisible by 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = list(range(6))\n",
    "fil_result = filter(lambda x: x % 2 == 0, numbers)\n",
    "list(fil_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reduce\n",
    "\n",
    "`reduce` is defined in the built-in `functools` module. Like `map` and `filter`, `reduce` takes two parameters, a function and an iterable. However, it returns a single value rather than another iterable. The way `reduce` works is to apply operations successively so that the example below effectively first sums 2 and 3 to make 5, then 5 and 5 to make 10, then 10 and 15 to make 25, and, finally, 25 and 20 to make the final result of 45."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import reduce\n",
    "\n",
    "numbers = [2, 3, 5, 15, 20]\n",
    "\n",
    "reduce(lambda x, y: x + y, numbers)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "interpreter": {
   "hash": "671f4d32165728098ed6607f79d86bfe6b725b450a30021a55936f1af379a247"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 64-bit ('codeforecon': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
